Deep learning approach to mitigate the cold-start problem in textual items recommendations

ABSTRACT

A method for mitigating cold starts in recommendations includes receiving a request that identifies a requested page and identifying a content vector of the requested page. The content vector is generated based on providing text of the requested page to a neural network text encoder. The method further includes selecting, based on the content vector, a link to a cold start page that does not satisfy a threshold level of interaction data. The selected link is ranked above a second link to a warm page that does satisfy the threshold level of the interaction data. The method further includes presenting the requested page with the selected link.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/699,545, filed Nov. 29, 2019, which is incorporated by referenceherein.

BACKGROUND

Websites serve pages to users who are looking for information. As usersattempt to find pages with pertinent information, such users may clickon multiple links, user interaction data (also referred to asinteraction data or click data) is generated. To assist users in his orher search, websites may provide users links to recommended pages. Suchlinks may be recommended based on the interaction data. Different pageson the system have different amounts of interaction data. For instance,new pages, which are referred to as cold start pages, may have little ifany interaction data when they are first added to the website. Thus,there is a challenge in recommending pages that include pertinentinformation to a currently presented page but for which there isinsufficient interaction data.

SUMMARY

In general, in one aspect, one or more embodiments relate to a methodthat includes receiving a request that identifies a requested page andidentifying a content vector of the requested page. The content vectoris generated based on providing text of the requested page to a neuralnetwork text encoder. The method further includes selecting, based onthe content vector, a link to a cold start page that does not satisfy athreshold level of interaction data. The selected link is ranked above asecond link to a warm page that does satisfy the threshold level of theinteraction data. The method further includes presenting the requestedpage with the selected link.

In general, in one aspect, one or more embodiments relate to a systemthat includes a processor and a memory coupled to the processor. Thememory includes a server application. The server application executes onthe processor and is configured for receiving a request that identifiesa requested page and identifying a content vector of the requested page.The content vector is generated based on providing text of the requestedpage to a neural network text encoder. The server application is furtherconfigured for selecting, based on the content vector, a link to a coldstart page. The cold start page does not satisfy a threshold level ofinteraction data. The selected link is ranked above a second link to awarm page that does satisfy the threshold level of the interaction data.The server application is further configured for presenting therequested page with the selected link.

In general, in one aspect, one or more embodiments relate to a methodthat includes identifying a requested page and identifying a contentvector of the requested page. The content vector is generated based onproviding text of the requested page to a neural network text encoder.The method further includes selecting, based on the content vector, alink to a cold start page that does not satisfy a threshold level ofinteraction data. The selected link is ranked above a second link to awarm page that does satisfy the threshold level of the interaction data.The method further includes displaying the requested page with theselected link.

Other aspects of the invention will be apparent from the followingdescription and the appended claims.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1A, FIG. 1B, and FIG. 1C show diagrams of systems in accordancewith disclosed embodiments.

FIG. 2A and FIG. 2B show flowcharts in accordance with disclosedembodiments.

FIG. 3 shows examples in accordance with disclosed embodiments.

FIG. 4A and FIG. 4B show computing systems in accordance with disclosedembodiments.

DETAILED DESCRIPTION

Specific embodiments of the invention will now be described in detailwith reference to the accompanying figures. Like elements in the variousfigures are denoted by like reference numerals for consistency.

In the following detailed description of embodiments of the invention,numerous specific details are set forth in order to provide a morethorough understanding of the invention. However, it will be apparent toone of ordinary skill in the art that the invention may be practicedwithout these specific details. In other instances, well-known featureshave not been described in detail to avoid unnecessarily complicatingthe description.

Throughout the application, ordinal numbers (e.g., first, second, third,etc.) may be used as an adjective for an element (i.e., any noun in theapplication). The use of ordinal numbers is not to imply or create anyparticular ordering of the elements nor to limit any element to beingonly a single element unless expressly disclosed, such as by the use ofthe terms “before”, “after”, “single”, and other such terminology.Rather, the use of ordinal numbers is to distinguish between theelements. By way of an example, a first element is distinct from asecond element, and the first element may encompass more than oneelement and succeed (or precede) the second element in an ordering ofelements.

In general, embodiments of the invention provide links to recommendedpages (referred to as cold start pages) for which there is insufficientuser interaction data. The links to the cold start pages are providedbased on a ranking function that is trained with user interaction data(referred to as interaction data) and then updated. The rank function isupdated based on the output of a text encoder. The updated rank functionmay rank cold start pages above warm pages so that newly added pages tothe system that are relevant to the requested page may be identified andrecommended.

A page may be a cold start page or a warm page based on whether there isa sufficient amount of interaction data for the page. The sufficiency ofinteraction data for a page may be based on a combination of a number ofclicks selecting the page, the length of time a page has been availableon the system (referred to as the age of the page), and an amount ofcross links to the page from other pages on the system. For example, apage that has been newly added to the system may not have any clicks,may not have been available from the system for a long enough period oftime, and may not have any cross links from other pages on the system.The new page may be referred to as a cold start page, which is incontrast to a warm page which has been clicked, has been available fromthe system, and is included in cross links from other pages.

In response to receiving a request, a server application identifies therequested page and transmits the content of the requested page. Thecontent of the requested page is transmitted with recommended links thatmay include cold start links to cold start pages ranked above warm pageson the system.

FIG. 1A, FIG. 1B, and FIG. 1C show diagrams of embodiments that are inaccordance with the disclosure to mitigate cold starts in textual itemrecommendations. The embodiments of FIG. 1A, FIG. 1B, and FIG. 1C may becombined and may include or be included within the features andembodiments described in the other figures of the application. Thefeatures and elements of FIG. 1A, FIG. 1B, and FIG. 1C are, individuallyand as a combination, improvements to technology computing systems,including computing systems that provide links to recommended pages. Thevarious elements, systems, and components shown in FIG. 1A, FIG. 1B, andFIG. 1C may be omitted, repeated, combined, and/or altered as shown fromFIG. 1A, FIG. 1B, and FIG. 1C. Accordingly, the scope of the presentdisclosure should not be considered limited to the specific arrangementsshown in FIG. 1A, FIG. 1B, and FIG. 1C.

Turning to FIG. 1A, the training application (104) is set of programsrunning on the server (102) (shown in FIG. 1C). The training application(104) trains the rank function (108) using the pages (152) and (154).The pages (152) and (154) are associated with the interaction data (124)(shown in FIG. 1C). The interaction data (124) identifies the clicksmade from a source page to a target page. A target page with a highernumber of clicks from a source page may be ranked higher than adifferent page with a fewer number of clicks from the same source page.

The pages (152) are distinct from the pages (154) based on the amount ofinteraction data for the individual pages. The pages (152) may bereferred to as warm pages and the pages (154) may be referred to as coldstart pages. For a warm page, a threshold amount of interaction data hasbeen acquired. For a cold start page, a threshold amount of interactiondata has not been acquired. The threshold amount of interaction data maybe based on an amount of interaction data, an age of the page, and anamount of cross links to the page from other pages on the system (100).

A warm page may satisfy the threshold amount of interaction data byhaving a threshold amount of interaction data, a threshold age, and/or athreshold amount of cross links. A cold start page may not satisfy thethreshold level of interaction data by not having the threshold amountof interaction data, the threshold age, and the threshold amount ofcross links.

The amount of interaction data for a page may include the number oftimes a page was clicked on from any previous page. A threshold amountof interaction data based on the amount of interaction data may besatisfied by a predetermined number of clicks (e.g., 10,000 clicks) or apercentage (e.g., 0.01 percent) of total clicks present in theinteraction data (124).

The age of the page may identify lengths of time related to the creationof a page, the publication or release date of a page, or the last updateto the page. A threshold amount of interaction data based on the age ofa page may be satisfied when a predetermined number of days has passedsince one of the dates related to the page. For example, the thresholdmay be for 30 days from the publication date of a page.

The amount of cross links may identify the number of links from otherpages to a target page. A threshold amount of interaction data based onthe amount of cross links may be satisfied by a predetermined number ofcross links to a target page or a percentage. For example, a page may bea warm page when there are, e.g., 1,000 cross links from other pages tothe page. As another example, a page may be a warm page when 0.1 percentof the pages of the system include a cross link to the page.

Additionally, the threshold amount of interaction data required todifferentiate between warm pages (152) and cold start pages (154) may bea combination of thresholds. For example, to be a warm page using acombination of thresholds, a page may need to have been published for atleast 30 days and have been clicked on at least 2,000 times.

As another example, to be a warm page, a page may need to have beenclicked on at least one time. Otherwise, the page may be a cold startpage.

For the pages (152) and (154), the training application (104) maygenerate identification vectors, which include the identification vector(130). The training application may also apply the text encoder (106) tothe pages (152) and (154) to generate the content vectors (132).

The identification vectors may be referred to as one hot vectors thatidentify individual pages. The identification vector (130) may have asmany elements as there are pages stored by the system (100), which maybe the sum total of pages (152) and (154). One element of the one hotvector is set to “1” and all other elements are set to “0” so that eachpage is associated with a different element of an identification vector.

The text encoder (106) generates a content vector from a page. As anexample, the text encoder (106) may take the first 30 words of the titleof a page and convert those words to word vectors, which are one hotvectors that identify the words used by the pages. The word vectors fromthe title may be combined to form the content vector (132). Combiningthe word vectors may be done by passing the word vectors through anencoder neural network (e.g., a convolutional neural network (CNN), arecurrent neural network (RNN), and attention network, etc.). Forexample, word2vec algorithms may be used as either a continuous bag ofwords or as a skip gram. The encoder network may be pretrained, may betrained using the pages of the system (100), or both.

The training application (104) generates sets of rank inputs and labelsfor the rank function (108). The rank input (156) may include a pair ofa source page and a target page with the label (162) that identifieswhether the target page was clicked on from the source page. In thepair, the source page may be identified by the content vector generatedfrom the source page and the target page may be identified with theidentification vector for the target page. The rank input (156) and thelabel (162) may include multiple pairs with labels. For example, therank input (156) may include 11 source page and target page pairs with11 labels. One of the 11 labels may identify one of the source page andtarget page pairs as being clicked on with the remaining 10 labelsidentifying source page and target page pairs that were not clicked on.

The rank function (108) takes the rank input (156) and generates therank output (160). The rank function (108) may operate on a singlesource page and target page pair by applying the S matrix (158) to thepair. For example, the identification vector of the target page may bemultiplied by the S matrix, with that product being multiplied by thecontent vector of the source page to generate a single value for thepair using the S matrix. The S matrix (158) may have a number of rowsequal to the number of pages (152) and (154) and a number of columnsequal to the number elements in the content vector (132).

The rank output (160) is the output generated by the rank function (108)from the rank input (156). When the rank input (156) includes multiplepairs, the rank output (160) may include multiple elements, to which asoftmax function is applied to identify one of the input pairs as thepair of a source and target pages that were clicked on. Extending theexample using 11 source page and target page pairs, the rank output(160) may be an 11 element vector with one dimension for each of theinputs from the rank input (156). Applying the softmax function to the11 element vector converts manipulates the values of the 11 elementvector so that the value of one of the elements dominates the remainingvalues to identify the source target pair that was clicked on.

The rank output (160) is compared to the label (162) by the lossfunction (164). The loss function calculates the error between the rankoutput (160) and the label (162) and feeds the error back to the Smatrix (158) so that future rank outputs generated with the rankfunction (108) will more closely match the label (162).

After training the rank function (108) to generate the S matrix (158),the training application generates the F matrix (168) with the F matrixgenerator (166). The F matrix generator (166) assembles the F matrixfrom the content vectors generated by the text encoder (106). Each rowof the F matrix (168) is one of the content vectors generated by thetext encoder (106) for the pages (152) and (154) of the system (100). Inthis manner, the S matrix (158) and the F matrix (168) may have the samenumber of elements and dimensions. The F matrix (168) is added to the Smatrix (158) to generate the updated S matrix (172) (shown in FIG. 1B).Updating the S matrix with the F matrix changes the ranking andrecommendation values output by the rank function (108) so that a coldstart page may be ranked above a warm page.

Turning to FIG. 1B, the server application (110) uses the updated Smatrix to generate page links, including the page link (174). The serverapplication (110) may service a request for the page (126) (shown inFIG. 1C) from which the content vector (132) was generated using thetext encoder (106) (shown in FIGS. 1A and 1C).

The rank engine (112) of the server application (110) includes the rankfunction (108) with the updated S matrix (172) that is trained by thetraining application (104) (shown in FIG. 1A). The rank engine (112)takes the content vector (132) as an input and outputs the recommendedpage links, including the page link (174). The page link (174) is arecommended link to a page of the system (100).

The recommended links may be generated by multiplying the content vector(132) by the updated S matrix (172) to generate a recommendation vectorwith the same number elements as the identification vectors. The valuesof the elements of the recommendation vector identify the strength ofthe recommendation for pages associated with the elements. The elementswith the highest values have the strongest recommendations and a number(e.g., 5) of the highest values may be used to identify the number ofrecommended pages. For example, if the first element of therecommendation vector has the highest value, the link to the pageassociated with the first element of the recommendation vector may bepresented as the top recommendation.

Turning to FIG. 1C, the system (100) includes the server (102), therepository (122), the developer device (134), and the user device (138).The server (102) may include the training application (104) and theserver application (110).

The training application (104) is a program on the server (102). Thetraining application (104) includes the text encoder (106) and the rankfunction (108). The training application (104) trains the rank function(108) using content vectors generated by the text encoder (106). Thetraining application (104) may be operated or controlled by thedeveloper device (134) with the developer application (136).

The server application (110) is a program on the server (102). Theserver application (110) includes the rank engine (112). The serverapplication (110) services requests from the user device (138) andtransmits pages, which may include the page (126) from the repository(122), with recommended links, which may include the link (128) from therepository (122), to the user device (138). The recommended links aregenerated with the rank engine (112).

The server (102) is an embodiment of the computing system (400) and thenodes (422) and (424) of FIG. 4A and FIG. 4B. The server (102) may beone of a set of virtual machines hosted by a cloud services provider todeploy the training application (104) and the server application (110)for a financial services provider.

The repository (122) is a computing system that may include multiplecomputing devices in accordance with the computing system (400) and thenodes (422) and (424) described below in FIGS. 4A and 4B. The repository(122) may be hosted by a cloud services provider for the financialservices provider. The cloud services provider may provide hosting,virtualization, and data storage services as well as other cloudservices and the financial services provider may operate and control thedata, programs, and applications that store and retrieve data from therepository. The data in the repository (122) may be processed byprograms executing on the server (102). The repository (122) may behosted by the same cloud services provider as the server (102). Therepository (122) may store the interaction data (124), pages (includingthe page (126)), links (including the link (128)), identificationvectors (including the identification vector (130)), and content vectors(including the content vector (132)).

The interaction data (124) is a recording of user interactions with thepages stored in the repository (122). The interaction data (124) mayinclude data related to mouse movement events, mouse click events, andkeyboard press events. The interaction data (124) may be referred to asclick data. The interaction data (124) identifies which pages have beenclicked on. The interaction data (124) may identify pairs of sourcepages and target pages. A source page is the page that includes the linkthat was clicked on. A target page is the page referenced by a link thatwas clicked on from a source page.

The pages stored on the repository (122) include the page (126). Thepages may be web pages that are served by the server application (110)in response to requests from the user application (140).

The links stored on the repository (122) include the link (128). A linkmay be a uniform resource identifier (URI), uniform resource locator(URL), hyperlink, etc. that identifies one of the pages stored on therepository (122). The link (128) may identify the page (126).

The identification vectors stored on the repository (122) include theidentification vector (130). An identification vector may be a vectorthat identifies a page with a particular element. Each identificationvector may have a number of dimensions equal to the number of pagesstored by the system (100) in the repository (122). The identificationvector (130) may be the identification vector of the page (126).

The content vectors stored on the repository (122) include the contentvector (132). A content vector is generated from a page using a textencoder and is related to the meaning of the content from within thepage using a number of latent feature dimensions. The number of latentfeature dimensions may be less than the number of dimensions for theidentification vectors.

The developer device (134) is an embodiment of the computing system(400) and the nodes (422) and (424) of FIG. 4A and FIG. 4B. Thedeveloper device (134) includes the developer application (136) foraccessing the training application (104). The developer application(136) may include a graphical user interface for interacting with thetraining application (104) to control training of the rank function(108).

The user device (138) is an embodiment of the computing system (400) andthe nodes (422) and (424) of FIG. 4A and FIG. 4B. The user device (138)includes the user application (140) for accessing the server application(110). The user application (140) may include a graphical user interfacefor interacting with the server application (110) to display pages andlinks from the repository (122). A user may operate the user application(140) to search for pages that answer questions about using the softwareof the financial services provider. The results may be displayed by theuser device (138) in the user application (140). The user device (138)may be operated by a customer of the financial services provider.

The developer application (136) and the user application (140) may beweb browsers that access the training application (104) and the serverapplication (110) using web pages hosted by the server (102). Thedeveloper application (136) and the user application (140) mayadditionally be web services that communicate with the trainingapplication (104) and the server application (110) usingrepresentational state transfer application programming interfaces(RESTful APIs). Although FIG. 1C shows a client server architecture, oneor more parts of the training application (104) and the serverapplication (110) may be local applications on the developer device(134) and the user device (138) without departing from the claimedscope.

FIG. 2A and FIG. 2B show flowcharts of the process (200) and the process(250) in accordance with the disclosure. The embodiments of FIGS. 2A and2B may be combined and may include or be included within the featuresand embodiments described in the other figures of the application. Thefeatures of FIGS. 2A and 2B are, individually and as an orderedcombination, improvements to the technology of computing systems. Whilethe various steps in the flowcharts are presented and describedsequentially, one of ordinary skill will appreciate that at least someof the steps may be executed in different orders, may be combined oromitted, and at least some of the steps may be executed in parallel.Furthermore, the steps may be performed actively or passively. Forexample, some steps may be performed using polling or be interruptdriven. By way of an example, determination steps may not have aprocessor process an instruction unless an interrupt is received tosignify that condition exists. As another example, determinations may beperformed by performing a test, such as checking a data value to testwhether the value is consistent with the tested condition.

Turning to FIG. 2A, the process (200) uses a trained and updated rankfunction to present cold start links. In Step 202, a request is receivedfor a requested page. The request may be received by a serverapplication from a user application and identify a page stored on arepository that is accessible by the server application. The request maybe a hypertext transfer protocol (HTTP) “GET” request that identifiesthe page with a uniform resource identifier (URI).

In Step 204, a content vector is identified for a request page. Thecontent vector may be identified by generating the content vector fromthe page or by looking up the content vector from a lookup table ordatabase that identifies pages and corresponding content vectorsgenerated from the pages. The content vector may be generated fromcontent of the requested page and the content may include text that isinput to a neural network text encoder to generate the content vector.For example, the content may include the first 10 words of a title ofthe page that are input to a neural network text encoder.

In Step 206, a link for a cold start page is ranked and selected above awarm page. The link may be selected with the content vector in responseto inputting the content vector to a rank function of a rank engine of aserver application. The output from the rank engine identifies and ranksa number of recommended links to pages. As an example, the input to therank function may be the content vector of the requested page and theoutput of the rank function may be an identification vector where thevalues for the dimensions the identification vector indicate thestrength of the recommendation of a link to a page associated with thedimension of the identification vector. A number (e.g., 5) of the pageswith the highest values may be identified as recommended pages and thelinks to those pages selected as recommended links to be presented withthe requested page.

The recommended links include the link to a cold start page (alsoreferred to as a cold start link) that does not satisfy a thresholdlevel of interaction data. The cold start link may be ranked above asecond link to a warm page that does satisfy the threshold level ofinteraction data based on the updated S matrix used by the rankfunction. The cold start link may be selected with the content vector byusing a rank engine. The rank engine may be trained using theinteraction data and a plurality of pages including the cold start page,the warm page and the requested page, which is further described withFIG. 2B. The threshold level of interaction data that differentiatesbetween cold pages and warm pages is selected from one or a combinationof an amount of interaction data, an age of the page, and an amount ofcross links to the page from other pages on the system.

In Step 208, the requested page is presented with a link. The requestedpage may be presented with the link, which is to the cold start pagethat does not satisfy the threshold level of interaction data.Presenting the requested page with the link may include generating atransmission page that includes the content from the requested page,includes the link within a set of recommended links, is transmitted fromthe server application to the user application, and is displayed by theuser application on the user device.

Turning to FIG. 2B, the process (250) trains a rank function. In Step252, content vectors are generated from warm and cold pages. The systemgenerates a content vector for each of the pages that may be served bythe system in response to requests. The pages of the system include thewarm pages that satisfy the threshold level of interaction data and coldstart pages that do not satisfy the threshold level of interaction data.

A content vector may be generated by identifying the title of a page andconverting the words from the title to word vectors. The wordsidentified from the title may be limited to a number of words (e.g., thefirst 20 words from the title). The word vectors may be input to aneural network text encoder that may include convolutional neuralnetworks, long short term memories, attention mechanisms, etc., whichgenerate the content vector from the word vectors.

In Step 254, a rank function is trained with a first matrix usinginteraction data and content vectors. The first matrix may be referredto as an S matrix that includes a first dimension of elements for thenumber of pages on the system and a second dimension of elements forlatent features. The latent features are features that are automaticallylearned by the system and correspond to the elements of the contentvectors. As an example, if the system has 10,000 pages and 50 latentfeatures, the S matrix may have 10,000 rows for the first dimension and50 columns for the second dimension. The number of latent features maybe selected by a developer of the system.

The rank function is trained by a training application. The rankfunction generates a rank output from a rank input. The rank output maybe compared to a label that corresponds to the rank input by a lossfunction. The loss function determines the error between the label andthe rank output. The S matrix is updated based on the error determinedby the loss function.

The training application may generate the rank input, which may includea source vector and a target vector. The rank input may be labeled witha label. The source vector is one of the content vectors generated witha text encoder and that corresponds to a source page, which is one ofthe pages on the system. The target vector may be an identificationvector that identifies a target page on the system. The label identifieswhether the target page was clicked on from the source page. The labelmay use a “1” to identify that the target page was clicked on andselected from the source page and a “0” otherwise.

In Step 256, a second matrix is generated from the first matrix and thecontent vectors. The second matrix may be referred to as the F matrixand has the same dimensions as the S matrix used by the rank function.By having the same dimensions, the F matrix and the S matrix may havethe same number of elements in each dimension. The F matrix may begenerated from the content vectors generated from the pages of thesystem. A row of the F matrix may be a content vector of a page of thesystem.

In Step 258, the first matrix is updated with the second matrix.Updating the first matrix (the S matrix) with the second matrix (the Fmatrix) generates an updated first matrix (also referred to as anupdated S matrix) that increases the rank of a cold start page above therank of a warm page. The first matrix (the S matrix) may be updated byadding the second matrix (the F matrix) to the first matrix, as shown inthe equation below.S=S+F  (Eq. 1)Addition of these matrices is possible since the S matrix and the Fmatrix have the same dimensions with the number rows equal to the numberpages on the system and the number of columns equal to the number ofelements (also referred to as latent features) used by the contextvectors. Adding the S matrix to the F matrix generates an updated Smatrix that ranks pages differently from the original S matrix. Whereasthe original S matrix may not rank a cold start page above a warm pagedue to the lack of interaction data, the updated S matrix may rank acold start page above a warm page even though the cold start page doesnot meet the threshold level of interaction data.

In Step 260, a requested page is presented with a link to a cold startpage ranked above a warm page. The requested page is presented with alink to the cold start page even though the cold start page does notsatisfy the threshold level of interaction data. Presenting therequested page with the link may include generating a transmission pagethat includes the content from the requested page, includes the linkwithin a set of recommended links, is transmitted from the serverapplication to the user application, and is displayed by a userapplication on the user device.

FIG. 3 shows examples of systems and interfaces in accordance with thedisclosure. The embodiments of FIG. 3 may be combined and may include orbe included within the features and embodiments described in the otherfigures of the application. The features and elements of FIG. 3 are,individually and as a combination, improvements to the technology oflink recommendation computing systems. The various features, elements,widgets, components, and interfaces shown in FIG. 3 may be omitted,repeated, combined, and/or altered as shown. Accordingly, the scope ofthe present disclosure should not be considered limited to the specificarrangements shown in FIG. 3 .

The user application (352) sends the request (304) to the serverapplication (302). The request (304) may include a uniform resourceidentifier (URI) to identify a page from the system that a user hasselected to view with the user application (352).

The server application (302) receives the request (304). The serverapplication (302) identifies the requested page (308) as the page beingrequested by the request (304) with the URI from the request (304).

The server application (302) identifies the content vector (306) as thecontent vector that corresponds to the requested page (308). The contentvector may have been previously generated from the text of the title(“Manually add transactions in QuickBooks Self-Employed”) of therequested page (308) by a text encoder and stored in a database to whichthe server application has access.

Recommended links, including the page link (312), are identified for thecontent vector (306). The recommended links may be generated on demand(i.e., in response to receiving the request 304) or may be generatedprior to receiving the request (304) and stored. For example, therecommended links for each page of the system may be generated inresponse to when the pages of the system are updated. The rank engine(310) may identify a number (e.g., 4) of highest recommended pages basedon the content vector (306), from which the recommended links aregenerated.

The server application (302) may generate a transmission page thatincludes the content from the requested page (308) and the page link(312) of the recommended links. The server application (302) transmitsthe transmission page with the content from the requested page (308) andthe recommended links to the user application (352).

The user application (352) receives and displays the transmission pagein the browser (354). The browser (354) displays the section (356) withcontent from the requested page (308) and displays the section (354)that includes the recommended links with the page link (312). Forexample, the page link (312) may be to the page titled “Add oldertransactions to QuickBooks Self-Employed”, which is the firstrecommended link. The page link (312) may be a cold start link to a coldstart page for which the interaction data is below the threshold level,in contrast to the other recommended links in the section (358), whichmay include interaction data that is above the threshold level. Bypresenting the cold start link, even though the cold start linkreferences a cold start page with insufficient interaction data, thesystem may reduce the number of clicks it takes for a user to findpertinent information.

Embodiments of the invention may be implemented on a computing systemspecifically designed to achieve an improved technological result. Whenimplemented in a computing system, the features and elements of thedisclosure provide a significant technological advancement overcomputing systems that do not implement the features and elements of thedisclosure. Any combination of mobile, desktop, server, router, switch,embedded device, or other types of hardware may be improved by includingthe features and elements described in the disclosure. For example, asshown in FIG. 4A, the computing system (400) may include one or morecomputer processors (402), non-persistent storage (404) (e.g., volatilememory, such as random access memory (RAM), cache memory), persistentstorage (406) (e.g., a hard disk, an optical drive such as a compactdisk (CD) drive or digital versatile disk (DVD) drive, a flash memory,etc.), a communication interface (412) (e.g., Bluetooth interface,infrared interface, network interface, optical interface, etc.), andnumerous other elements and functionalities that implement the featuresand elements of the disclosure.

The computer processor(s) (402) may be an integrated circuit forprocessing instructions. For example, the computer processor(s) may beone or more cores or micro-cores of a processor. The computing system(400) may also include one or more input devices (410), such as atouchscreen, keyboard, mouse, microphone, touchpad, electronic pen, orany other type of input device.

The communication interface (412) may include an integrated circuit forconnecting the computing system (400) to a network (not shown) (e.g., alocal area network (LAN), a wide area network (WAN) such as theInternet, mobile network, or any other type of network) and/or toanother device, such as another computing device.

Further, the computing system (400) may include one or more outputdevices (408), such as a screen (e.g., a liquid crystal display (LCD), aplasma display, touchscreen, cathode ray tube (CRT) monitor, projector,or other display device), a printer, external storage, or any otheroutput device. One or more of the output devices may be the same ordifferent from the input device(s). The input and output device(s) maybe locally or remotely connected to the computer processor(s) (402),non-persistent storage (404), and persistent storage (406). Manydifferent types of computing systems exist, and the aforementioned inputand output device(s) may take other forms.

Software instructions in the form of computer readable program code toperform embodiments of the invention may be stored, in whole or in part,temporarily or permanently, on a non-transitory computer readable mediumsuch as a CD, DVD, storage device, a diskette, a tape, flash memory,physical memory, or any other computer readable storage medium.Specifically, the software instructions may correspond to computerreadable program code that, when executed by a processor(s), isconfigured to perform one or more embodiments of the invention.

The computing system (400) in FIG. 4A may be connected to or be a partof a network. For example, as shown in FIG. 4B, the network (420) mayinclude multiple nodes (e.g., node X (422), node Y (424)). Each node maycorrespond to a computing system, such as the computing system shown inFIG. 4A, or a group of nodes combined may correspond to the computingsystem shown in FIG. 4A. By way of an example, embodiments of theinvention may be implemented on a node of a distributed system that isconnected to other nodes. By way of another example, embodiments of theinvention may be implemented on a distributed computing system havingmultiple nodes, where each portion of the invention may be located on adifferent node within the distributed computing system. Further, one ormore elements of the aforementioned computing system (400) may belocated at a remote location and connected to the other elements over anetwork.

Although not shown in FIG. 4B, the node may correspond to a blade in aserver chassis that is connected to other nodes via a backplane. By wayof another example, the node may correspond to a server in a datacenter. By way of another example, the node may correspond to a computerprocessor or micro-core of a computer processor with shared memoryand/or resources.

The nodes (e.g., node X (422), node Y (424)) in the network (420) may beconfigured to provide services for a client device (426). For example,the nodes may be part of a cloud computing system. The nodes may includefunctionality to receive requests from the client device (426) andtransmit responses to the client device (426). The client device (426)may be a computing system, such as the computing system shown in FIG.4A. Further, the client device (426) may include and/or perform all or aportion of one or more embodiments of the invention.

The computing system or group of computing systems described in FIGS. 4Aand 4B may include functionality to perform a variety of operationsdisclosed herein. For example, the computing system(s) may performcommunication between processes on the same or different system. Avariety of mechanisms, employing some form of active or passivecommunication, may facilitate the exchange of data between processes onthe same device. Examples representative of these inter-processcommunications include, but are not limited to, the implementation of afile, a signal, a socket, a message queue, a pipeline, a semaphore,shared memory, message passing, and a memory-mapped file. Furtherdetails pertaining to a couple of these non-limiting examples areprovided below.

Based on the client-server networking model, sockets may serve asinterfaces or communication channel end-points enabling bidirectionaldata transfer between processes on the same device. Foremost, followingthe client-server networking model, a server process (e.g., a processthat provides data) may create a first socket object. Next, the serverprocess binds the first socket object, thereby associating the firstsocket object with a unique name and/or address. After creating andbinding the first socket object, the server process then waits andlistens for incoming connection requests from one or more clientprocesses (e.g., processes that seek data). At this point, when a clientprocess wishes to obtain data from a server process, the client processstarts by creating a second socket object. The client process thenproceeds to generate a connection request that includes at least thesecond socket object and the unique name and/or address associated withthe first socket object. The client process then transmits theconnection request to the server process. Depending on availability, theserver process may accept the connection request, establishing acommunication channel with the client process, or the server process,busy in handling other operations, may queue the connection request in abuffer until server process is ready. An established connection informsthe client process that communications may commence. In response, theclient process may generate a data request specifying the data that theclient process wishes to obtain. The data request is subsequentlytransmitted to the server process. Upon receiving the data request, theserver process analyzes the request and gathers the requested data.Finally, the server process then generates a reply including at leastthe requested data and transmits the reply to the client process. Thedata may be transferred, more commonly, as datagrams or a stream ofcharacters (e.g., bytes).

Shared memory refers to the allocation of virtual memory space in orderto substantiate a mechanism for which data may be communicated and/oraccessed by multiple processes. In implementing shared memory, aninitializing process first creates a shareable segment in persistent ornon-persistent storage. Post creation, the initializing process thenmounts the shareable segment, subsequently mapping the shareable segmentinto the address space associated with the initializing process.Following the mounting, the initializing process proceeds to identifyand grant access permission to one or more authorized processes that mayalso write and read data to and from the shareable segment. Changes madeto the data in the shareable segment by one process may immediatelyaffect other processes, which are also linked to the shareable segment.Further, when one of the authorized processes accesses the shareablesegment, the shareable segment maps to the address space of thatauthorized process. Often, only one authorized process may mount theshareable segment, other than the initializing process, at any giventime.

Other techniques may be used to share data, such as the various datadescribed in the present application, between processes withoutdeparting from the scope of the invention. The processes may be part ofthe same or different application and may execute on the same ordifferent computing system.

Rather than or in addition to sharing data between processes, thecomputing system performing one or more embodiments of the invention mayinclude functionality to receive data from a user. For example, in oneor more embodiments, a user may submit data via a graphical userinterface (GUI) on the user device. Data may be submitted via thegraphical user interface by a user selecting one or more graphical userinterface widgets or inserting text and other data into graphical userinterface widgets using a touchpad, a keyboard, a mouse, or any otherinput device. In response to selecting a particular item, informationregarding the particular item may be obtained from persistent ornon-persistent storage by the computer processor. Upon selection of theitem by the user, the contents of the obtained data regarding theparticular item may be displayed on the user device in response to theuser's selection.

By way of another example, a request to obtain data regarding theparticular item may be sent to a server operatively connected to theuser device through a network. For example, the user may select auniform resource locator (URL) link within a web client of the userdevice, thereby initiating a Hypertext Transfer Protocol (HTTP) or otherprotocol request being sent to the network host associated with the URL.In response to the request, the server may extract the data regardingthe particular selected item and send the data to the device thatinitiated the request. Once the user device has received the dataregarding the particular item, the contents of the received dataregarding the particular item may be displayed on the user device inresponse to the user's selection. Further to the above example, the datareceived from the server after selecting the URL link may provide a webpage in Hyper Text Markup Language (HTML) that may be rendered by theweb client and displayed on the user device.

Once data is obtained, such as by using techniques described above orfrom storage, the computing system, in performing one or moreembodiments of the invention, may extract one or more data items fromthe obtained data. For example, the extraction may be performed asfollows by the computing system in FIG. 4A. First, the organizingpattern (e.g., grammar, schema, layout) of the data is determined, whichmay be based on one or more of the following: position (e.g., bit orcolumn position, Nth token in a data stream, etc.), attribute (where theattribute is associated with one or more values), or a hierarchical/treestructure (consisting of layers of nodes at different levels ofdetail-such as in nested packet headers or nested document sections).Then, the raw, unprocessed stream of data symbols is parsed, in thecontext of the organizing pattern, into a stream (or layered structure)of tokens (where each token may have an associated token “type”).

Next, extraction criteria are used to extract one or more data itemsfrom the token stream or structure, where the extraction criteria areprocessed according to the organizing pattern to extract one or moretokens (or nodes from a layered structure). For position-based data, thetoken(s) at the position(s) identified by the extraction criteria areextracted. For attribute/value-based data, the token(s) and/or node(s)associated with the attribute(s) satisfying the extraction criteria areextracted. For hierarchical/layered data, the token(s) associated withthe node(s) matching the extraction criteria are extracted. Theextraction criteria may be as simple as an identifier string or may be aquery presented to a structured data repository (where the datarepository may be organized according to a database schema or dataformat, such as XML).

The extracted data may be used for further processing by the computingsystem. For example, the computing system of FIG. 4A, while performingone or more embodiments of the invention, may perform data comparison.Data comparison may be used to compare two or more data values (e.g., A,B). For example, one or more embodiments may determine whether A>B, A=B,A !=B, A<B, etc. The comparison may be performed by submitting A, B, andan opcode specifying an operation related to the comparison into anarithmetic logic unit (ALU) (i.e., circuitry that performs arithmeticand/or bitwise logical operations on the two data values). The ALUoutputs the numerical result of the operation and/or one or more statusflags related to the numerical result. For example, the status flags mayindicate whether the numerical result is a positive number, a negativenumber, zero, etc. By selecting the proper opcode and then reading thenumerical results and/or status flags, the comparison may be executed.For example, in order to determine if A>B, B may be subtracted from A(i.e., A−B), and the status flags may be read to determine if the resultis positive (i.e., if A>B, then A−B>0). In one or more embodiments, Bmay be considered a threshold, and A is deemed to satisfy the thresholdif A=B or if A>B, as determined using the ALU. In one or moreembodiments of the invention, A and B may be vectors, and comparing Awith B requires comparing the first element of vector A with the firstelement of vector B, the second element of vector A with the secondelement of vector B, etc. In one or more embodiments, if A and B arestrings, the binary values of the strings may be compared.

The computing system in FIG. 4A may implement and/or be connected to adata repository. For example, one type of data repository is a database.A database is a collection of information configured for ease of dataretrieval, modification, re-organization, and deletion. DatabaseManagement System (DBMS) is a software application that provides aninterface for users to define, create, query, update, or administerdatabases.

The user, or software application, may submit a statement or query intothe DBMS. Then the DBMS interprets the statement. The statement may be aselect statement to request information, update statement, createstatement, delete statement, etc.

Moreover, the statement may include parameters that specify data, ordata container (database, table, record, column, view, etc.),identifier(s), conditions (comparison operators), functions (e.g. join,full join, count, average, etc.), sort (e.g. ascending, descending), orothers. The DBMS may execute the statement. For example, the DBMS mayaccess a memory buffer, a reference or index a file for read, write,deletion, or any combination thereof, for responding to the statement.The DBMS may load the data from persistent or non-persistent storage andperform computations to respond to the query. The DBMS may return theresult(s) to the user or software application.

The computing system of FIG. 4A may include functionality to present rawand/or processed data, such as results of comparisons and otherprocessing. For example, presenting data may be accomplished throughvarious presenting methods. Specifically, data may be presented througha user interface provided by a computing device. The user interface mayinclude a GUI that displays information on a display device, such as acomputer monitor or a touchscreen on a handheld computer device. The GUImay include various GUI widgets that organize what data is shown as wellas how data is presented to a user. Furthermore, the GUI may presentdata directly to the user, e.g., data presented as actual data valuesthrough text, or rendered by the computing device into a visualrepresentation of the data, such as through visualizing a data model.

For example, a GUI may first obtain a notification from a softwareapplication requesting that a particular data object be presented withinthe GUI. Next, the GUI may determine a data object type associated withthe particular data object, e.g., by obtaining data from a dataattribute within the data object that identifies the data object type.Then, the GUI may determine any rules designated for displaying thatdata object type, e.g., rules specified by a software framework for adata object class or according to any local parameters defined by theGUI for presenting that data object type. Finally, the GUI may obtaindata values from the particular data object and render a visualrepresentation of the data values within a display device according tothe designated rules for that data object type.

Data may also be presented through various audio methods. In particular,data may be rendered into an audio format and presented as sound throughone or more speakers operably connected to a computing device.

Data may also be presented to a user through haptic methods. Forexample, haptic methods may include vibrations or other physical signalsgenerated by the computing system. For example, data may be presented toa user using a vibration generated by a handheld computer device with apredefined duration and intensity of the vibration to communicate thedata.

The above description of functions presents only a few examples offunctions performed by the computing system of FIG. 4A and the nodesand/or client device in FIG. 4B. Other functions may be performed usingone or more embodiments of the invention.

While the invention has been described with respect to a limited numberof embodiments, those skilled in the art, having benefit of thisdisclosure, will appreciate that other embodiments can be devised whichdo not depart from the scope of the invention as disclosed herein.Accordingly, the scope of the invention should be limited only by theattached claims.

What is claimed is:
 1. A method comprising: receiving a request thatidentifies a requested page; identifying a content vector of therequested page, the content vector generated based on providing text ofthe requested page to a neural network text encoder; selecting, based onthe content vector, a link to a cold start page, the cold start page notsatisfying a threshold level of interaction data, the selected linkranked above a second link to a warm page that does satisfy thethreshold level of the interaction data; and presenting the requestedpage with the selected link.
 2. The method of claim 1, furthercomprising: generating the content vector of a plurality of contentvectors from the cold start page of a plurality of pages; training arank function using the interaction data, the rank function including afirst matrix, the first matrix having first matrix dimensions, and afirst dimension of the first matrix dimensions having a plurality offirst elements for the plurality of pages and a second dimension of thefirst matrix dimensions having a plurality of second elements for aplurality of latent features corresponding to elements of the contentvector; generating a second matrix having second matrix dimensions, thesecond matrix dimensions being the same as the first matrix dimensions,and the second matrix generated from the plurality of content vectors;and updating the first matrix with the second matrix to generate anupdated first matrix that increases a rank of the cold start page abovethe warm page.
 3. The method of claim 1, wherein the interaction dataand a plurality of pages including the cold start page, the warm page,and the requested page are used to train a rank function, wherein thethreshold level of the interaction data is one selected from a groupconsisting of an amount of the interaction data, an age of the page, andan amount of cross links to the page from other pages of the pluralityof pages, wherein the warm page satisfies the threshold level ofinteraction data by including at least one of a threshold amount ofinteraction data, a threshold age, and a threshold amount of crosslinks, and wherein the cold start page does not include at least one ofthe threshold amount of interaction data, the threshold age, and thethreshold amount of cross links and does not satisfy the threshold levelof interaction data.
 4. The method of claim 1, further comprising:training a rank function by: generating a rank output from a rank inputwith the rank function, comparing the rank output to a label with a lossfunction, and updating a first matrix of the rank function based on thecomparison of the rank output to the label prior to updating the firstmatrix with a second matrix generated from a plurality of contentvectors including the content vector.
 5. The method of claim 1, furthercomprising: training a rank function by: generating a rank input for therank function, the rank input comprising a source vector and a targetvector, the rank input labeled with a label, the source vector being oneof a plurality of content vectors identifying a source page, theplurality of content vectors including the content vector, the targetvector being an identification vector identifying a target page, and thelabel identifying whether the target page was clicked on from the sourcepage.
 6. The method of claim 1, further comprising: generating thecontent vector by identifying a title of the cold start page, andconverting a set of words from the title to word vectors.
 7. The methodof claim 1, wherein the neural network text encoder comprises one ormore from a group consisting of a convolutional neural network, a longshort term memory, and an attention mechanism.
 8. A system comprising: aprocessor; a memory coupled to the processor; and the memory comprisinga server application, wherein the server application executes on theprocessor and is configured for: receiving a request that identifies arequested page; identifying a content vector of the requested page, thecontent vector generated based on providing text of the requested pageto a neural network text encoder; selecting, based on the contentvector, a link to a cold start page, the cold start page not satisfyinga threshold level of interaction data, the selected link ranked above asecond link to a warm page that does satisfy the threshold level of theinteraction data; and presenting the requested page with the selectedlink.
 9. The system of claim 8, wherein the memory further comprises atraining application, wherein the training application executes on theprocessor and is configured for: generating the content vector of aplurality of content vectors from the cold start page of a plurality ofpages; training a rank function using the interaction data, the rankfunction including a first matrix, the first matrix having first matrixdimensions, and a first dimension of the first matrix dimensions havinga plurality of first elements for the plurality of pages and a seconddimension of the first matrix dimensions having a plurality of secondelements for a plurality of latent features corresponding to elements ofthe content vector; generating a second matrix having second matrixdimensions, the second matrix dimensions being the same as the firstmatrix dimensions, and the second matrix generated from the plurality ofcontent vectors; and updating the first matrix with the second matrix togenerate an updated first matrix that increases a rank of the cold startpage above the warm page.
 10. The system of claim 9, wherein thetraining application is further configured for: training a rank functionby: generating a rank output from a rank input with the rank function,comparing the rank output to a label with a loss function, and updatinga first matrix of the rank function based on the comparison of the rankoutput to the label prior to updating the first matrix with a secondmatrix.
 11. The system of claim 9, wherein the training application isfurther configured for: training a rank function by: generating a rankinput for the rank function, the rank input comprising a source vectorand a target vector, the rank input labeled with a label, the sourcevector being one of a plurality of content vectors identifying a sourcepage, the plurality of content vectors including the content vector, thetarget vector being an identification vector identifying a target page,and the label identifying whether the target page was clicked on fromthe source page.
 12. The system of claim 8, wherein the serverapplication is further configured for: generating the content vector by:identifying a title of the cold start page, converting a set of wordsfrom the title to word vectors, and generating the content vector byinputting the word vectors to a neural network text encoder.
 13. Thesystem of claim 12, wherein the neural network text encoder comprisesone or more from a group consisting of a convolutional neural network, along short term memory, and an attention mechanism.
 14. The system ofclaim 8, further comprising: a repository that stores the interactiondata, a plurality of pages, a plurality of content vectors including thecontent vector, and a plurality of identification vectors.
 15. A methodcomprising: identifying a requested page; identifying a content vectorof the requested page, the content vector generated based on providingtext of the requested page to a neural network text encoder; selecting,based on the content vector, a link to a cold start page, the cold startpage not satisfying a threshold level of interaction data, the selectedlink ranked above a second link to a warm page that does satisfy thethreshold level of the interaction data; and displaying the requestedpage with the selected link.
 16. The method of claim 15, wherein theinteraction data and the plurality of pages including the cold startpage, the warm page, and a requested page, wherein the threshold levelof interaction data is one selected from a group consisting of an amountof interaction data, an age of the page, and an amount of cross links tothe page from other pages of the plurality of pages, wherein the warmpage satisfies the threshold level of interaction data by including atleast one of a threshold amount of interaction data, a threshold age,and a threshold amount of cross links, and wherein the cold start pagedoes not include at least one of the threshold amount of interactiondata, the threshold age, and the threshold amount of cross links anddoes not satisfy the threshold level of interaction data.
 17. The methodof claim 15, further comprising: training a rank function by: generatinga rank output from a rank input with the rank function, comparing therank output to a label with a loss function, and updating a first matrixof the rank function based on the comparison of the rank output to thelabel prior to updating the first matrix with a second matrix generatedfrom a plurality of content vectors including the content vector. 18.The method of claim 15, further comprising: training a rank function by:generating a rank input for the rank function, the rank input comprisinga source vector and a target vector, the rank input labeled with alabel, the source vector being one of a plurality of content vectorsidentifying a source page, the plurality of content vectors includingthe content vector, the target vector being an identification vectoridentifying a target page, and the label identifying whether the targetpage was clicked on from the source page.
 19. The method of claim 15,further comprising: generating the content vector by identifying a titleof the cold start page, and converting a set of words from the title toword vectors.
 20. The method of claim 15, further comprising: generatingthe content vector generated from content of the cold start page,wherein the content comprises text that is input to a neural networktext encoder to generate the content vector, and wherein the neuralnetwork text encoder comprises one or more from a group consisting of aconvolutional neural network, a long short term memory, and an attentionmechanism; and presenting a requested page with a link to the cold startpage.